Multilayer Perceptron Experiments on Sine Wave

We generated synthetic data using sin(x) function, which we aim to learn. The blue colors are the training set which we will use to train our model. The orange colors are the test set which we will use to check if our model can generalize and did learn the sine function

Our inputs are the x-values (horizontal axis) and our targets are y = sin(x), which we will train our model to predict given x.

We show an animation below of the neural network parameters (MLP) and the evolution of how it fits the data over its training (epochs). We find that while the network fits the training set well, it failed to extend the sine wave function in the test set.

Once Loop Reflect

While I was initially surprised, I realized that making this highly unbiased MLP model generalize a sine function may be quite hard since there are an infinite possible functions that fit the same training set. There is no rule that tells the model that it has to extend the periodicity infinitely in both directions. In fact, this fitted model is just as valid as any of the functions that fit the data, from the perspective of the model.

In other words, we will need to incorporate our assumption that the periodicity extends infinitely, by introducing some inductive bias into the model, such that it is encouraged to find solutions that have periodicity.

Still, watching how the neural network learns to fit on the training data is quite fun!

Acknowledgements

Thanks Daniel Stanley Tan for the reference code!